First Virtual support for FTP servers
*************************************

This document describes how to support current information databases on FTP
servers with minimal modifications to server software and no modifications to
client software. For information on supporting your World Wide Web server,
please see our home page at http://www.fv.com 

An FTP site that already exists can be upgraded to use the First
Virtual protocols with relative ease, particularly if all information
on that FTP server is going to be assumed to be owned by the same
entity (e.g., for cost recovery). If someone connects and logs in with
the "anonymous" login, then only free information is readable;
however, it is expected that directory listings of for-pay information
will still be permitted, in support of Archie and similar systems. If
someone connects and logs in with the ID of "fvftp" then the password
provided should be a valid First Virtual Account ID, which can be
checked at the time of the login against First Virtual's server and
perhaps against a list of server-specific "deadbeats". Future
retrievals during this login would be billed to that account.

Required Software
=================

To support billing for FTP downloads, you will need first to obtain the First
Virtual API code from ftp://ftp.fv.com/pub/code/fv-api. You can also obtain
patches for FTP servers from ftp://ftp.fv.com/pub/code/fv-ftpd 

Implementation
++++++++++++++

Initially, one may be tempted to use the fv program as is, passing
command-line arguments as necessary. However, there are two problems
with this scheme. The first problem is that the .$ files contain what
some may consider sensitive information that should not be transmitted
to users connecting anonymously. The second is a problem of overhead:
on a heavily-loaded server, an extra two or three popen() calls with
their associated shell and execution overheads may be undesirable. To
this end, the above-described programs have been written as a main()
for a library of routines. There is a routine to initialize and
uninitialize internal data structures (including parsing of .$ files),
calculating costs of a file, and billing an account for the cost of a
file. Most of the underlying machinery (in terms of socket management,
SGCP protocol management, and so on) is also exposed. Using this
library instead of executing separate programs has three advantages:
first, the SGCP socket (if any) is left open between calls, which is
more efficient than reestablishing it for each operation; second, the
overhead of executing separate programs is eliminated, as is the
problems of formatting arguments such that the shell does not corrupt
them; third, popen() need not be rewritten to not use a shell which is
unavailable due to an earlier chroot() call. However, there are some
disadvantages as well. Since the routines execute in the same process
as the FTP server daemon, it is not possible to use suid flags to make
the .$ files visible to the billing routines without being visible to
those downloading files.

Because the .$ files should not be visible to those downloading files
(since otherwise someone could cause inconvenience to the server), the
following scheme is used. All .$ files are owned by the user
"ftpadmin," which is distinct from the user "ftp". They are made
readable only to the owner and not writable to anyone. Files in the
upload areas are not readable once created. The ~ftp/bin/ls command is
copied from FV_ls and suid to ftpadmin, which allows it to read the .$
files. Since the anonymous ftp process cannot read these files, it
cannot deliver them even accidentally. On the other hand, it cannot
charge for them either. There are two solutions to this problem. The
first would be to put fv into ~ftp/bin and suid to ftpadmin, spawning
separate processes to do the calculations in the same way FV_ls
does. The second method is to read all the .$ files in the anonymous
area into memory before doing the chroot() call and changing the
effective user id of the daemon process. Since scanning all the
directories might be a time-consuming process, one would periodically
do this scan, accumulate the contents of all .$ files into one file,
and store that file outside of the anonymous FTP area accessible after
the chroot() call or in a file readable only by ftpadmin. If this file
is stored in a standard format, the initialization routines can
automatically load it into memory and retain it, reinitializing
current values from the appropriate parts of memory when needed. The
primary dangers here are bloating of the multiple FTP processes with
.$ information they will likely never use and having the individual .$
files out of synchronization with the collected .$ file. For servers
which are primarily download servers (i.e., those without much upload
activity) or servers with few .$ files, these dangers are
lessened. This latter approach is used in the sample server. The shell
script to rebuild the collected .$ files is fv_regen, which calls
fv_gen_preload.

Building the system
+++++++++++++++++++

Make the appropriate modifications to your FTP server. Basically, the
following is required: 

Where users log in, add a condition to not only check for "ftp" or
"anonymous", but also "fvftp". If the user logs in as "fvftp", call fv_preload()
and fv_init() with the right file names, accept a password, and use fv_check()
to verify its accuracy. If the password is acceptable ("active" is good, "new" is
probably good, and you may want to keep a deadbeat list yourself) then finish
logging in the user as you would for anonymous, but additionally set a flag
that says this is a First Virtual login, and remember the password. Before
sending information, call fv_costof() with the full path name of the file to
determine whether it costs money. If not, continue as normal. If so, make sure
that the First Virtual flag is set, or deny the download with an informative error
message. If the download completes, call fv_chargefor() to bill the user. It's
that simple. 

In our sample server, we have also added "This costs money" messages in
several places and inserted a five-second delay before starting transmission of
for-fee files. In addition, we have replaced the "ls" program that normally
resides in ~ftp/bin with one copied from the "fv_ls.c" program, which you must
compile and link (don't forget static linking). This program uses two routines,
"glob" and "realpath", which are linked in from the wu-ftp 2.1c FTP server.
They glob file names and discover the full path of a file given a relative path.
If you are using a WU variant, you should compile your ftpd, then compile and
link fv_ls, using realpath.o and glob.o from the ftpd directory. Otherwise,
replace these routines. 

The next step is to parameterize how much you want to charge. If there
is only one account to be creditted, it is probably best to modify
fv_vars.c and compile in the server name, seller, and so on. If you
want multiple sellers, you should probably give each a separate
directory. In any case, the "fv_gen_preload" program should be
run. This program goes through the FTP directory collecting all the .$
files and putting their contents into one "preload" file. This preload
file is loaded into memory by fv_preload() and consulted whenever a .$
file is asked to be loaded by fv_chargefor() or fv_costof(). The
preload program should be rerun any time a .$ file changes, including
creation and deletion. The rationale behind this is to keep users from
downloading .$ files, which may be considered to contain information a
seller might not want an anonymous user downloading. Hence, the .$
files in the anonymous FTP directory should not be readable by the uid
the anonymous fptd process is using. Generally, they are owned by
"ftpadmin" under the group "ftp". All the directories and files in the
anonymous area are owned by "ftpadmin" and group "ftp". They should
generally be group readable, except for the .$ files, which should be
only readable by "ftpadmin". The "preload" file (output by the
fv_gen_preload program) should be outside the area accessible after
the root has been changed with chroot() and readable by the ftpd
process before it changes to user "ftp". Finally, the fv_ls program
(which should probably be called "ls" in the ~ftp/bin directory, just
to stay compatible with non-anonymous FTP connections) should be
setuid and owned by "ftpadmin" in order to allow it to read the .$
files while showing directory listings with prices.